Geometry and Determinism of Optimal Stationary Control in Partially Observable Markov Decision Processes

It is well known that any finite state Markov decision process (MDP) has a deterministic memoryless policy that maximizes the discounted longterm expected reward. Hence for such MDPs the optimal control problem can be solved over the set of memoryless deterministic policies. In the case of partially observable Markov decision processes (POMDPs), where there is uncertainty about the world state, optimal policies must generally be stochastic, if no additional information is presented, e.g., the observation history. In the context of embodied artificial intelligence and systems design an agent’s policy underlies hard physical constraints and must be as efficient as possible. Having this in mind, we focus on memoryless POMDPs. We cast the optimization problem as a constrained linear optimization problem and develop a corresponding geometric framework. We show that any POMDP has an optimal memoryless policy of limited stochasticity, which means that we can give an upper bound to the number of deterministic policies that need to mixed to obtain an optimal stationary policy, regardless of the specific reward function.